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Outline 

Stennis Space Center 

• Background 

• Sources of error in geopositional assessment 

• Error model 

• Discussion of geopositional error computation methods 

• Modeled performance of geopositional error computation 
methods 

• Conclusions and recommendations 
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Background 
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• 1947 - U.S. Bureau of the Budget. National Map Accuracy Standards. 

- Establishes equivalent of circular error criteria as error standard of maps of various scales. 

1962 - Clyde Greenwalt and Melvin Shultz. Principles of Error Theory and Cartographic 
Applications. 

- Provides rigorous treatment of circular error assuming that error is 

• Zero mean (no horizontal bias) • Normally distributed ^ • Near-circular 

• 1963 - Melvin Shultz. Circular Error Probability of a Quantity Affected by a Bias. 

- Provides limited treatment of error with horizontal bias. 

• 1 990 - MIL-STD-600001 . Mapping, Charting and Geodesy Accuracy. 

- Adopts the 1963 Shultz approach to horizontal jDias. % Discusses empirical approach as an 
alternative estimate. 

• 1998 - Federal Geographic Data Committee. National Standard for Spatial Data 
Accuracy (NSSDA).~ 

- Adopts Greenwalt and Shultz approach, but swaps RMSE for standard deviation. No provision 
for horizontal bias. 

• 2003 - Joseph McCollum (USFS). Map Error and Root Mean Square. 

- Paper calls Greenwalt and Shultz method into question. 

• 2003 - USGS Proposal for Revision of NSSDA. 

- Out of Geography Discipline. POC: John Conroy, iconrov@usqs.gov . 

• 2004 (first version 2002?) - Tom Ager (NIMA InnoVision). An Analysis of Metric 
Accuracy Definitions and Methods of Computation. 

- White paper supports empirical approach. Also modifies Shultz approach to provide for large 
horizontal bias. 
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Revision Status 
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The revision of the NSSDA standard is currently in step 4, or the draft 
stage, of the 12-step FGDC standards approval process 
(http://www.fqdc.gov/standards/directives/dir1.htmn . 

• Progress on the standard development will continue based on 
funding priorities. v 


Proposal Stage 

Step 1 , Develop Proposal 

Step 2, Review Proposal 

Project Stage 

Step 3, Set Up Project 

Draft Stage 

Step 4, Produce Working Draft 

Step 5, Review Working Draft 

Review Stage 

Step 6, Review and Evaluate Committee Draft 

Step 7, Approve Standard for Public Review 

Step 8, Coordinate Public Review 

Step 9, Respond to Public Comments 

Step 10, Evaluate Responsiveness to Public Comments 

Step 1 1 , Approve Standard for Endorsement 

Final Stage 

Step 12, Endorsement 
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Sources of Error in Geopositional 

Assessment 
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Assessment Error 

- Ground Control Error 

• Pointing 

• Measurement 

- Analyst Error 

• Pointing 

Product Error (potential) 

- Spatial Resolution 

- Pointing (Displacement) 

- Azimuth 

- Scale 

- Orthogonality 

- Other product distortion 

- Terrain effects 


•\ 


• “Pointing error” for surveyors & analysts is here intended 
to mean the errors these individuals have in picking their 
target. 


>• random error \ 


• “Measurement error” for ground control is here intended 
to mean the error inherent in the measuring instrument or 
.. r- system (GPS in this case). 


• constant systematic error 

> \ • “Pointing error” for a geo-imaging system is here in- 

tended to mean the constant separation between esti- 
mated target coordinates and actual target coordinates. 

y • functional systematic error 

J 
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Check Point Error 
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Check Point Error - differences between 
image and reference coordinates 


Reference 

Coordinate 


AX -X -X , 

z image, i reference ,i 

&Y =Y. -Y f 

i image, i reference ,i 


Check point error radial 
magnitude calculated by 

A R t = ^AA,. 2 + AY' 2 
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Error Component Estimates 
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The error model chosen for generalized assessment 


y 

v image 


X + e 


where 


8 = 8 


cons tan t 


+ 8 


zero-mean 


• Horizontal Bias - an estimate of the constant error, designated here 
as /v H is the magnitude of the vector sum of the average error in the X 
and the Y 

<u H =J(jxf+( ay) 


• Circular Standard Error - an estimate of the zero-mean circular 
equivalent error valid even for elliptical error distributions with 
minimum to maximum error ratios as low as 0.6 


(7 c — 


^ AX ® AY 


where o a = 


l! 


X(aa-,-ay) 2 & fe^-Ar) 2 


n - 1 


O AY ~ 


1 


n — 1 


Tom Ager used the horizontal error defined on the right, 17 2 T 

but Greenwalt and Shultz found this to be invalid for a H = +<j ay , 

minimum to maximum error ratios less than 0.8. V 2 
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RMSE Definitions 

— Sterinis Space Center 

• RMSE - Root mean square error (horizontal bias & zero- 
mean error not decoupled) 

- ID , — T | T 

RMSE <=PL^~ RMSE , = JX 

- 2D (NSSDA General) 

RMSE r = ^RMSE x 2 + RMSE/ 

- 2D (NSSDA Case 2*) 

RMSE C = 0.5 * (RMSE x + RMSE y ) 

* RMSE C is a recasting of terms in formula from NSSDA 
Appendix Case 2. It is not found explicitly in the NSSDA. 
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Circular Error Definitions 
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• CE 90 - The radial error which 90% of all errors in a circular distribution 
will not exceed (adapted from Greenwalt and Shultz, 1962) 

- Equivalent to the Circular Map Accuracy Standard (CMAS) 

• CE 95 - The radial error which 95% of all errors in a circular distribution 
will not exceed (adapted from Greenwalt and Shultz, 1962) 

- Equivalent to Accuracy,- (from NSSDA) 

• In the normal case, circular error may be generally defined as the 

circle radius, R, that satisfies the conditions of the equation below 
(where C.L is the desired confidence level ); however, there is no 
analytical solution to this equation. 


C.L. = 


• R 


-R 






2 ™x°y^-p 2 ) 


exp 


-1 

2(1 - P 2 ) 


\ 2 

X ~Px 

G X J 


"2 P 




V v_// ^ 


v y 


y-p. 


\ G y J 


+ 


y-Rx 


c- 


v ^ j 


dydx 
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Common CEgo Estimates 
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• RMSE based (NSSDA) 

- Appendix A: General ► CE 90 = 1.5175 RMSE r 

- Appendix A: Case 2 ► CE 90 = 2.1460* RMSE c 






Bias and Standard Circular 
Error based 

Sum of squares — 

- Shultz approach 

accounting for bias 

- Ager approach 
accounting for bias _► 
(modified Shultz) 


CE 90 = J(2.1460-<r c ) 2 +//„ 2 

2 3 

CE % = 2.1272<7 c +0.1674//,, +0.3623— — 0.055^ 

<7 C <T C 

When pl h Io c < 0.1 CE 90 = 2.1460<J C 

When 0.1 < n H lo c ^ 3 apply equation from Shultz 


When ia h !o c > 3 CE 90 = 0.98 6ju h + 1 .4548<j c 


Empirically estimated 

- 90 th percentile ► CE 90 =90 th percentile of A R 

- Radial error for 1 st point of 
percentile rank > 90 
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Circular Error Modeling Study 

Stennis Space Center 

• Assumed bivariate normal distribution of errors 

• Modeled population ( all possible check points) as 1 M 

points 

• Modeled sample ( simulated target range) as 40 points 
(generated 10,000 trials of 40) 

• Constrained o c to 1 ( unitless for modeling purposes, but 

for spaceborne commercial imaging o c ~ 1 meter) 

~‘ v % 

• Varied o min lo max from 0 to 1 ( distributions from univariate 
through elliptical to perfectly circular) 

• Varied p H from 0 to 1 0,000 
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Example Trial 



Bias Direction = 45 


min 


max 


= 0.5 


f^H 


= 30 


Stennis Space Center 


60 


<D 

> 


40 


20 - 


0 


0 


“Errors” from 40 
simulated points 



Legend 

Simulated check point 
90% Error Ellipse 

ce 90 


20 


40 60 

X error 
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NSSDA RMSE r Based 

i — Stennis Space Center 


CE 90 =1.5175-RMSE r 

•.’ * 

' 
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CE go Estimate/Population CE= 


NSSDA RMSE r Based Confidence Interval 
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RMSE Based CE on 

r 90 

with 95% Cl 
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NSSDA Case 2 
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CE 90 =2.1460 RMSE c 
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CE go Estimate/Population CE| 



NSSDA Case 2 Confidence Interval (cardinal 

direction) 
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NSSDA Case 2 (bias direction 0 deg) CE go 

with 95% Cl 
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CE go Estimate/Population CE| 


NSSDA Case 2 Confidence Interval (45° off axis) 
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NSSDA Case 2 (bias direction 45 deg) CE go 

with 95% Cl 
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Sum of Squares 
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CE go Estimate/Population CE^ 


Sum of Squares Confidence Interval 
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Sum of Squares CE go 
with 95% Cl 
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CE go Estimate/Population CE^ 



Sum of Squares Results by Error Distribution 

Shape 

— — ^ ■ Stennis Space Center 


Sum of Squares CE go 

by cr . h 
J min max 



Range of typical 
values 

for spaceborne 

commercial 

imaging 


^H fo C 
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Ager Approach 

^ Sterinis Space Center 

• When Pfjoc ^ 0.1 
Cis 90 = 2.1460<J C 

V 



• When 0.1 < /i H lo c ^ 3 

- v” % 

CEg 0 = 2 . 1272 (J c + 0.1 674jU h + 0.3623 




0.055 



• When PfJoQ > 3 

CE 90 =0.98 6jU h +1.4548<7 c 
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CE go Estimate/Population CE^ ( 
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Ager Approach Confidence Interval 


1.6 


1.4 


1.2 
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Ager Approach CE 
with 95% Cl 


90 


^=3 


Note the steps at 0.1 and 3. Beyond a bias to 
zero-mean error ratio of 3, the Shultz curve took a 
sharp negative turn due to the highest order term. 









X. 1 f 
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CE go Estimate/Population CE^ 




Ager Approach Results by Error Distribution 

Shape 
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Ager Approach CE go 
by a . h 

J min max 


o 



^H /C C 
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“Empirical” Approach 
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• Given radial error magnitude calculated by 

AR,=jAXr+AY , 2 

v 

CE go = 90 th percentile of A R 
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CE go Estimate/Population CE^ 


“Empirical” Approach Confidence Interval 
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Empirical CE gQ 
with 95% Cl 


1.6r 



0 6 1 i 1 * 1 i i 

0 5 10 15 20 25 30 
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CE go Estimate/Population CE^ ( 



“Empirical” Approach Results by Error 

Distribution Shape 
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CE go Estimate/Population CE^ 0 CE go Estimate/Population C^ Q 


Side-by-Side Summary 
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Ager modification to Shultz method 
has little estimation bias and less 
uncertainty than empirical. 
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Conclusions and Recommendations 
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• RMSE based methods distort circular error estimates (up 
to 50% overestimation). 

• The empirical approach is the only statistically unbiased 

estimator offered. ' 

• Ager modification to Shultz approach is nearly unbiased, 

but cumbersome. - - 

• All methods hover around 20% uncertainty (@ 95% 
confidence) for low geopositional bias error estimates. This 

requires careful consideration in assessment of higher 
accuracy products. 
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